Management system and management method for computer system

ABSTRACT

This management system for a computer system including a plurality of elements for which performance values are to be monitored generates one or more plans on the basis of: a performance value history of a specific bottleneck (an element that has been identified as the cause of a detected failure); management information that includes information indicating, for each countermeasure, the relation between a plurality of countermeasure attributes and a plurality of bottlenecks; and a countermeasure attribute corresponding to the specific bottleneck, and displays the one or more plans thus generated. Each of the displayed one or more plans includes one or more countermeasures. In each of the displayed one or more plans, each of the one or more countermeasures is classified into one of a plurality of countermeasure types.

TECHNICAL FIELD

The present invention generally relates to management of a computersystem.

BACKGROUND ART

Aggregation of IT infrastructures, rise of a cloud environment, and thelike have made computer systems larger and more complicated. In such acomputer system, it is difficult to determine countermeasures against adetected failure. In the case where an administrator of the computersystem is a novice who has less knowledge or experience than an expert,it is particularly difficult to determine countermeasures.

For example, PTL 1 discloses the following technique, that is, atechnique of identifying a virtual logical volume which is a cause ofunsatisfaction of required performance, identifying consumption tendencyof capacity of a pool, and generating and displaying countermeasureswhich can be implemented after predetermined time on the basis of theidentified virtual logical volume and consumption tendency of thecapacity.

CITATION LIST Patent Literature

[PTL 1] WO 2014/073045

SUMMARY OF INVENTION Technical Problem

While, according to PTL 1, countermeasures are displayed, all thecountermeasures are the same type of countermeasures with which requiredperformance is expected to be satisfied. It is not always easy for anadministrator (particularly, a novice) to select a countermeasure amongthe countermeasures of the same type.

Solution to Problem

A management system of a computer system including a plurality ofelements whose performance values are monitored generates one or moreplans on the basis of history of a performance value of a specificbottleneck (an element identified as a cause of a detected failure),management information including information indicating relationshipbetween a plurality of countermeasure attributes and a plurality ofbottlenecks for each of a plurality of countermeasures, and acountermeasure attribute corresponding to the specific bottleneck, anddisplays the generated one or more plans. Each of the displayed one ormore plans includes one or more countermeasures. In each of thedisplayed one or more plans, each of the one or more countermeasures isclassified into one of a plurality of countermeasure types.

Advantageous Effects of Invention

In each of the displayed one or more plans, each of the one or morecountermeasures is classified into one of a plurality of countermeasuretypes. There is also a case where at least one plan includes combinationof two or more countermeasures. An administrator selects a plan from thedisplayed plans. In other words, the administrator selects acountermeasure in units of plan. Therefore, it is easy for theadministrator to select a countermeasure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates outline of an embodiment.

FIG. 2 illustrates a configuration of the whole system according to theembodiment.

FIG. 3 illustrates a configuration of a management computer.

FIG. 4 illustrates a configuration of failure countermeasureinformation.

FIG. 5 illustrates a configuration of countermeasure type information.

FIG. 6 illustrates a configuration of side effect information.

FIG. 7 illustrates a configuration of threshold information.

FIG. 8 illustrates a configuration of plan information.

FIG. 9 illustrates flow of the whole processing performed in theembodiment.

FIG. 10 illustrates flow of plan generation.

FIG. 11 illustrates flow of countermeasure type determination.

FIG. 12 illustrates flow of optimal combination generation.

FIG. 13 illustrates flow of emergency countermeasure selection.

FIG. 14 illustrates flow of screen display and plan selection.

FIG. 15 is an auxiliary view of explanation of countermeasure typedetermination.

FIG. 16 illustrates an example of case 1.

FIG. 17 illustrates an example of case 2.

FIG. 18 illustrates an example of case 3.

FIG. 19 illustrates an example of case 4.

FIG. 20 illustrates an example of case 5.

FIG. 21 illustrates an example of case 6.

FIG. 22 illustrates an example of case 7.

FIG. 23 is an auxiliary view of explanation of optimal combinationgeneration.

FIG. 24 is a first auxiliary view of explanation of emergencycountermeasure selection.

FIG. 25 is a second auxiliary view of explanation of emergencycountermeasure selection.

FIG. 26 is a third auxiliary view of explanation of emergencycountermeasure selection.

FIG. 27 is a fourth auxiliary view of explanation of emergencycountermeasure selection.

FIG. 28 is an auxiliary view of explanation of effectivenessrecalculation.

DESCRIPTION OF EMBODIMENT

An embodiment will be described below.

Note that, while, in the following description, there is a case whereprocessing is described using a “program” as a subject, because theprogram performs predetermined processing using storage resources (forexample, a memory) and/or a communication interface device (for example,a communication port), or the like, as appropriate by being executed bya processor (for example, a CPU (Central Processing Unit)), the subjectof the processing may be made the processor. The processing describedusing the program as a subject may be made processing performed by theprocessor or an apparatus having the processor. Further, the processormay include a hardware circuit which performs part or all of theprocessing. The program may be installed in each controller from aprogram source. The program source may be, for example, a programdistributing computer or a computer readable storage medium. Further, inthe following description, two or more programs may be implemented asone program, or one program may be implemented as two or more programs.

Further, in the following description, a management system may beconfigured with one or more computers. Specifically, for example, in thecase where the computer displays information (specifically, in the casewhere the computer displays information at a display device of thecomputer or the computer transmits information for display to a remotecomputer for display), the computer is the management system.

Further, in the following description, an “element” means a component ofthe computer system, and, specifically, a generic term of each of aplurality of nodes (apparatuses) constituting the computer system andeach of a plurality of components held by each node. A node includes aphysical node (for example, a server apparatus) and a logical node (forexample, program instance executed at the server apparatus). In asimilar manner, a component includes a physical component (for example,a CPU) and a logical component (for example, a logical volume). In thefollowing description, while number is used as identificationinformation of the element, other types of identification informationmay be used in place of or in addition to number.

In recent years, a computer system becomes larger and more complicatedbecause of, for example, at least one of the followings:

A scale of processing handled by the computer system becomes larger.

Like cloud service, a number of types of processing are executed by thecomputer system.

Types of nodes within the computer system increase.

Because as a result of internal configuration of a node becomingcomplicated, types of components constituting the node (for example,logical components and physical components) increase, it becomes furthernecessary to manage these components.

Because a virtualization technique (for example, virtualization of aserver, virtualization of a network, virtualization of a storage andvirtualization of a data center) becomes widespread, it becomes possibleto divide or aggregate apparatuses.

Deployment and migration techniques are advancing.

Here, “becoming larger” indicates that the number of elements to bemanaged within the computer system such as nodes constituting thecomputer system and components of the nodes increases. Further,“becoming complicated” indicates at least one of relationship betweenelements becoming M:1, 1:N or M:N (M and N are each integers of 2 ormore) by increase of types of elements to be managed, at least one ofvalues of M and N becoming greater and relationship between elementschanging from moment to moment.

In the case where a failure of the computer system is detected and acause element (an element as a cause of the failure (for example, a rootcause)) is identified through failure analysis such as RCA (Root CauseAnalysis), the management computer can present effective countermeasures(possible countermeasures). However, it is difficult for anadministrator (particularly, a novice) to select one countermeasure ortwo or more countermeasures to be combined from merely display of a listof effective countermeasures. Selection of a countermeasure is furtherdifficult in the computer system which has become larger andcomplicated.

Therefore, as illustrated in FIG. 1, in the present embodiment, themanagement computer 201 (an example of the management system) holdsmanagement information 360 which includes information indicatingrelationship between a plurality of countermeasure attributes associatedwith a countermeasure and a plurality of bottlenecks for each of aplurality of countermeasures defined in advance. Each of the “pluralityof bottlenecks” described here is an element which can be a cause (forexample, a root cause) of a failure which can occur. The managementcomputer 201 generates one or more plans on the basis of the managementinformation 360, history of a performance value of a specific bottleneckwhich is a bottleneck identified as a cause of the detected failure, anda countermeasure attribute corresponding to the specific bottleneckamong the plurality of countermeasure attributes of the plurality ofcountermeasures. The management computer 201 displays the generated oneor more plans. Each of the displayed one or more plans includes one ormore countermeasures.

In each of the displayed one or more plans, each of the one or morecountermeasures is classified into one of a plurality of countermeasuretypes. There is also a case where at least one plan includes combinationof two or more countermeasures. The administrator selects a desired planfrom the displayed plans. In other words, the administrator selects acountermeasure in units of plan. In other words, while a countermeasuretype is different depending on the specific bottleneck (the cause of thefailure) even for the same countermeasure, the administrator does nothave to determine a countermeasure type of each countermeasure inaccordance with the specific bottleneck (the cause of the failure),determine whether any two or more countermeasures can be combined ordetermine compatibility of the combination of the two or morecountermeasures. Therefore, it is easy for the administrator to select acountermeasure.

As illustrated in FIG. 1, in the case where a failure is detected, and abottleneck of the failure is identified through RCA (an example offailure analysis), plan generation is kicked. The management computer201 executes countermeasure type determination and optimal combinationgeneration which will be described later on the basis of the managementinformation 360 (for example, a countermeasure attribute correspondingto the specific bottleneck) and an amount of change of a performancevalue in accordance with the history of the performance value of thespecific bottleneck for each of the plurality of countermeasures. Bythis means, each countermeasure is classified into one of the pluralityof countermeasure types. The management computer 201 generates one ormore plans on the basis of the classification result.

The plurality of countermeasure attributes respectively associated withthe plurality of countermeasures include

-   (x1) change of a performance value which is expected for the    countermeasure, and-   (x2) at least one of start required time which is time required    until the countermeasure is started and execution required time    which is time from when the countermeasure is started until when the    countermeasure is finished. An example of (x1) is an evaluation    function 403 which will be described later (see FIG. 4). An example    of (x2) is start required time 404 and execution required time 407    which will be described later (see FIG. 4).

In the present embodiment, the plurality of countermeasure types includefundamental treatment and emergency treatment.

The “fundamental treatment” is a countermeasure for fundamentallysolving a failure. Specifically, for example, the fundamental treatmentis a countermeasure with which the performance value of the specificbottleneck is expected to be returned to equal to or smaller than afirst threshold of the performance value until time limit.

The “emergency treatment” is a countermeasure for temporarilysuppressing or preventing degradation of influence of a failure.Specifically, for example, the emergency treatment is a countermeasurewith which an amount of change of the performance value of the specificbottleneck is expected to be reduced.

In the description of the present embodiment, in order to avoid mixtureof the description, a countermeasure as a type will be referred to as“treatment”, a countermeasure classified into the fundamental treatmentwill be referred to as a “fundamental countermeasure”, and acountermeasure classified into the emergency treatment will be referredto as an “emergency countermeasure”.

Further, in the present embodiment, the plurality of countermeasuretypes further include follow-up treatment. The “follow-up treatment” isa countermeasure as follow-up. The “follow-up” is an example of processmonitoring, and is a countermeasure of monitoring whether a condition ofany element (resource) gets worse or whether a new failure occurs whilethe countermeasure is being implemented. A secondary effect of theperformance value is associated with at least one countermeasure as oneof the countermeasure attributes, and a follow-up countermeasure (acountermeasure classified into the follow-up treatment) is associatedwith at least one secondary effect. In the present embodiment, forconvenience sake, the secondary effect will be referred to as a “sideeffect”.

An example of display of the generated one or more plans is a planselection screen 150 illustrated in FIG. 1. According to the planselection screen 150, the generated one or more plans are arranged indescending order of priority. Each of the one or more plans includes oneor more countermeasures, and the one or more countermeasures is at leasta fundamental countermeasure among the emergency countermeasure, thefollow-up countermeasure and the fundamental countermeasure. Further,each of the one or more plans includes an extent of influence of a plan,required time and execution cost. The priority of each of the one ormore plans is calculated by the management computer 201 on the basis ofat least one of the number of countermeasures included in the plan, theextent (for example, the number of elements) of influence of the plan,the required time for the plan and the execution cost of the plan.

While terms such as the “emergency treatment” and the “side effect” aretypically used in a medical field, general idea in the medical fieldcannot be applied in management of the computer system, and, if suchapplication is possible, the management computer 201 according to thepresent embodiment cannot be constituted. An example of the reasons isas follows. In other words, in the computer system, a correspondingcountermeasure type is not fixed for at least one countermeasure amongthe plurality of countermeasures, and is different according toconditions (for example, at least one of which element is the specificbottleneck, an amount of change of the future performance value inaccordance with history of the performance value of the specificbottleneck, an effect expected as a result of execution of thecountermeasure (for example, an amount of change of the performancevalue of the specific bottleneck after the countermeasure isimplemented), the start required time 404 of the countermeasure and theexecution required time 407 of the countermeasure). For example, eventhe same countermeasure may become an emergency countermeasure in onecondition and may become a fundamental countermeasure in anothercondition. The management computer 201 classifies each of the pluralityof countermeasures (except, for example, a follow-up countermeasure)into any countermeasure type while taking into account the conditionwhen the plan is generated, and generates one or more plans on the basisof the classification result. In the generated plan, sometimes, only thefundamental countermeasure exists, and sometimes, combination of two ormore countermeasures, specifically, combination of the fundamentalcountermeasure and the follow-up countermeasure or combination of thefundamental countermeasure, the emergency countermeasure and thefollow-up countermeasure exists. Meanwhile, in the medical field,typically the type of the countermeasure is fixed (for example,hemostasis which is typically emergency treatment, does not becomefundamental treatment).

Details of the present embodiment will be described below.

FIG. 2 illustrates a configuration of the whole system according to theembodiment.

The management computer 201 is coupled to the computer system 200 via acommunication network (for example, a LAN (Local Area Network) or a WAN(Wide Area Network)) 204.

The computer system 200 includes a plurality of monitoring targetapparatuses 203. The monitoring target apparatus 203 is an example ofthe node. The monitoring target apparatus 203 has a plurality ofcomponents (computer resources). The plurality of components include,for example, a port, a network interface having a port, a MPB (MainProcessor Board) (an example of a processor) and a memory.

FIG. 3 illustrates a configuration of the management computer 201.

The management computer 201 has a network interface 302, an I/O(Input/Output) device 301, a memory 390 and a processor 303 coupled tothese.

The network interface 302 is coupled to the communication network 204.The I/O device 301 is a user interface device such as an input device(for example, a keyboard and a pointing device) and an output device(for example, a display device).

The memory 390 includes a computer program executed by the processor 303and information referred to or updated by the processor 303.

Examples of the computer program include, for example, an OS (Operatingsystem) 304, an AP (Application Program) 305, a failure detectionprogram 306, a bottleneck identification program 307, a countermeasuretype determination program 308, a trend prediction program 309, a plangeneration program 310, an optimal combination generation program 311, apriority calculation program 312, a screen display program 313, anemergency countermeasure selection program 323, an effectivenessrecalculation program 322 and a plan execution program 324. The failuredetection program 306 regularly acquires performance data of an element(for example, the monitoring target apparatus 203 and its component) ofthe computer system 200 and stores the acquired performance data inperformance information and detects a failure from the acquiredperformance data. The performance data for the element may include, forexample, a performance value (measurement value) and a measurement timepoint. The bottleneck identification program 307 identifies an elementas a cause of the detected failure on the basis of, for example,performance information and configuration information (for example,displays candidate elements of a root cause as a result of RCA andreceives selection of an element as the root cause from theadministrator (user)). The countermeasure type determination program 308determines a countermeasure type of each of a plurality ofcountermeasures (for example, countermeasures other than the follow-upcountermeasure) on the basis of the specific bottleneck (an elementidentified as a cause of a failure) and the management information 360.The trend prediction program 309 predicts trend (specifically, a futureperformance value of the specific bottleneck) on the basis of thehistory of the performance value of the specific bottleneck. The plangeneration program 310 generates one or more plans each including one ormore countermeasures. The optimal combination generation program 311generates combination of the fundamental countermeasure and theemergency countermeasure. The priority calculation program 312calculates priority of the generated plan. The screen display program313 displays one or more plans arranged according to the priority. Theemergency countermeasure selection program 323 selects the emergencycountermeasure to be combined with the fundamental countermeasure. Theeffectiveness recalculation program 322 recalculates effectiveness ofthe plan selected by the administrator from the displayed one or moreplans. The plan execution program 324 executes the plan selected by theadministrator. Among these programs, at least one of the failuredetection program 306 and the bottleneck identification program 307 maybe executed by a computer other than the management computer 201. Inother words, a computer other than the management computer 201 maycollect performance data (update the performance information 314),detect a failure and identify a bottleneck.

Examples of the information to be referred to or updated by theprocessor 303 include performance information 314, configurationinformation 315 and management information 360. The performanceinformation 314 includes performance data acquired for each of theplurality of elements (monitoring target elements). In other words, inthe performance information 314, performance data (including aperformance value) for each of the plurality of elements is accumulated.The configuration information 315 indicates a configuration (forexample, a topology configuration of a plurality of elements) of thecomputer system 200. The management information 360 includes informationnecessary for management of the computer system 200 other than theperformance information 314 and the configuration information 315.Specifically, for example, the management information 360 includesthreshold information 316, side effect information 317, countermeasuretype information 318, plan information 319, failure countermeasureinformation 320 and time limit information 321. The thresholdinformation 316 indicates a threshold of the performance value. The sideeffect information 317 indicates a side effect associated with thecountermeasure. The countermeasure type information 318 indicates aclassification result of the countermeasure. The plan information 319indicates the generated one or more plans. The failure countermeasureinformation 320 is an example of information indicating relationshipbetween a plurality of countermeasure attributes associated with thecountermeasure and a plurality of bottlenecks for each of the pluralityof countermeasures. The time limit information 321 indicates acceptabletime until the performance value of the specific bottleneck is recovered(for example, the performance value becomes equal to or smaller than afirst threshold), that is time limit.

FIG. 4 illustrates a configuration of the failure countermeasureinformation 320.

The failure countermeasure information 320 is, for example, a table, andhas a record for each of the plurality of countermeasures (for examples,countermeasures other than the follow-up countermeasure). Acountermeasure ID 401 and a countermeasure attribute set 450 are storedin each record. In each record, the countermeasure attribute set 450exists for each of a plurality of bottleneck IDs 402. The countermeasureID 401 is an ID of the countermeasure. The bottleneck ID 402 (forexample, “B001”) is an ID of the bottleneck (element).

The countermeasure attribute set 450 is a plurality of countermeasureattributes associated with the countermeasure. Examples of thecountermeasure attribute includes, for example, an evaluation function403, start required time 404, a side effect ID 405, an extent ofinfluence 406, execution required time 407, the number of times ofoperation 408 and execution cost 409. The evaluation function 403 is anexample of change of the performance value expected for thecountermeasure, and is a function for deriving an amount of change ofthe performance value expected after the countermeasure is implemented.The start required time 404 is time required for starting thecountermeasure, and, specifically, a function or a constant indicatingsuch time. The side effect ID 405 is an ID of the side effect. Theextent of influence 406 is an extent affected by the countermeasure,and, for example, the extent is expressed with an ID (for example, E002)of the element affected by the countermeasure. The execution requiredtime 407 is time from when the countermeasure is started until when thecountermeasure is finished. The number of times of operation 408 is therequired number of times of operation performed in execution of thecountermeasure. The execution cost 409 is cost required for executingthe countermeasure.

FIG. 5 illustrates a configuration of the countermeasure typeinformation 318.

The countermeasure type information 318 is, for example, a table, andhas a record for each of the plurality of effective countermeasures. The“effective countermeasure” is a countermeasure classified into acountermeasure type other than ineffective treatment as a result ofclassification of countermeasures. In each record, a failure ID 501, acountermeasure ID 502, countermeasure type name 503, necessity ofemergency countermeasure 504 and inclination after countermeasure 505are stored. The failure ID 501 is an ID of the detected failure. Thecountermeasure ID 502 is an ID of an effective countermeasure for thefailure. The countermeasure type name 503 indicates name of acountermeasure type (emergency treatment or fundamental treatment) ofthe effective countermeasure. The necessity of emergency countermeasure504 is effective for the fundamental countermeasure and indicateswhether or not combination with the emergency countermeasure isrequired. The inclination after countermeasure 505 is an example of anamount of change after the countermeasure is implemented, and indicatesinclination of the performance value expected by the countermeasure(specifically, inclination calculated on the basis of the evaluationfunction 403 corresponding to the countermeasure and the specificbottleneck and inclination of a linear function of the performancevalue).

FIG. 6 illustrates a configuration of the side effect information 317.

The side effect information 317 is, for example, a table and has arecord for each of a plurality of side effects. In each record, a sideeffect ID 601, side effect outline 602, side effect detail 603,follow-up outline 604 and a follow-up releasing trigger 605 are stored.The side effect ID 601 is an ID of the side effect. The side effectoutline 602 indicates outline of the side effect. The side effect detail603 indicates detail of the side effect, and, for example, indicates afunction for deriving a maximum width of degradation of the performancevalue. The follow-up outline 604 indicates outline of follow-up for theside effect. The follow-up releasing trigger 605 indicates a trigger forfinishing the follow-up.

FIG. 7 illustrates a configuration of the threshold information 316.

The threshold information 316 is, for example, a table, and has a recordfor each of the plurality of bottlenecks. In each record, a bottleneckID 701, a first threshold 702 and a second threshold 703 are stored. Thefirst threshold 702 is one of thresholds for the performance value ofthe bottleneck. The second threshold 703 is a threshold meaning that theperformance value is worse than the first threshold. In the case where alower performance value indicates better performance, the secondthreshold 703 is a value larger than the first threshold 702. On theother hand, in the case where a higher performance value indicatesbetter performance, the second threshold 703 is a value smaller than thefirst threshold 702. The number of thresholds may be one, or three ormore for the same bottleneck. Further, at least one performance valuemay be fixed or may be able to be changed by the administrator.

FIG. 8 illustrates a configuration of the plan information 319.

The plan information 319 is, for example, a table, and has a record foreach of one or more plans. In each record, a plan ID 801, a fundamentalcountermeasure ID 802, an emergency countermeasure ID 803 and afollow-up countermeasure ID 804 are stored. The plan ID 801 is an ID ofthe plan. The fundamental countermeasure ID 802 is a countermeasure IDof the fundamental countermeasure. The emergency countermeasure ID 803is a countermeasure ID of the emergency countermeasure. The follow-upcountermeasure ID 804 is a countermeasure ID of the follow-upcountermeasure. In the case where there is no emergency countermeasurein the plan, the emergency countermeasure ID 803 corresponding to theplan is “N/A” (Not Applicable). In the case where there is no follow-upcountermeasure in the plan, the follow-up countermeasure ID 804corresponding to the plan is also “N/A”.

Processing performed in the present embodiment will be described below.

FIG. 9 illustrates flow of the whole processing performed in the presentembodiment.

The failure detection program 306 detects a failure which has occurred,on the basis of performance data from the computer system 200 (S901).The bottleneck identification program 307 identifies a bottleneck of thedetected failure (S902). In the case where the detected failure is aperformance failure (for example, the performance value exceeds at leastthe first threshold) (S903: Yes), the plan generation program 310 iskicked.

The plan generation program 310 executes plan generation (S904). Theplan generation program 310 kicks the screen display program 313, andthe screen display program 313 displays a plan selection screen 150 inaccordance with the plan information 319 which is a result of the plangeneration (S905). In the plan selection screen 150, as illustrated inFIG. 1, one or more plans are displayed.

The screen display program 313 kicks the effectiveness recalculationprogram 322 in the case where selection of a plan among the one or moreplans displayed at the plan selection screen 150 is received from theadministrator (S906). The effectiveness recalculation program 322executes recalculation of effectiveness for the selected plan (S907).The meaning of recalculation of effectiveness is checking whether or notthe plan selected by the administrator is effective when selection isreceived from the administrator because time has elapsed since a plangeneration time point. “Recalculation” described here means performingprocessing similar to processing performed in plan generation (S904) forgenerating the selected plan, on the basis of a plan selection timepoint.

If the selected plan is not effective as a result of recalculation ofeffectiveness (S908: No), the effectiveness recalculation program 322may notify the screen display program 313 that the selected plan isineffective, and the screen display program 313 may display at the planselection screen 150 that the selected plan is ineffective. Display ofineffectiveness of the selected plan may be performed by hiding theselected plan and leaving only the effective plan at the plan selectionscreen 150.

If the selected plan remains effective as a result of recalculation ofeffectiveness (S908: Yes), the effectiveness recalculation program 322kicks the plan execution program 324. The plan execution program 324executes the plan selected by the administrator (S909).

The outline of the whole processing performed in the present embodimenthas been described above.

Plan generation (S904), screen display (S905) and recalculation ofeffectiveness (S907) will be described in detail below with reference toFIGS. 10 to 14. In the description, FIGS. 15 to 28 which visuallyillustrate the processing will be referred to as appropriate. Further,information (for example, functions and parameters) in an xy coordinatesystem indicated in FIGS. 15 to 28 exists in a memory 390. Further, inthe xy coordinate system, an x coordinate is a time point, and a ycoordinate is the performance value of the specific bottleneck (thebottleneck identified in S902).

FIG. 10 illustrates flow of the plan generation (S904).

The plan generation program 310 kicks the trend prediction program 309.The trend prediction program 309 identifies history of the performancevalue (history of a pair of the performance value and the measurementtime point) of the specific bottleneck from the performance information314 and predicts trend 1501 (see FIG. 15) in accordance with the historyof the performance value (S1001). In the present embodiment, the historyof the performance value and the trend 1501 are linear functions of theperformance value and, thus, an amount of change of the performancevalue is inclination of the linear function.

The plan generation program 310 kicks the countermeasure typedetermination program 308. The countermeasure type determination program308 executes countermeasure type determination (S1002). Thecountermeasure type determination is determining a countermeasure typeof each of the plurality of countermeasures (for example,countermeasures other than the follow-up countermeasure), that is,determining which of the countermeasure types each of the plurality ofcountermeasures corresponds to in the failure detected in S901.

The plan generation program 310 kicks the optimal combination generationprogram 311. The optimal combination generation program 311 executesoptimal combination generation (S1003). The optimal combinationgeneration includes determining whether or not the emergencycountermeasure is necessary for the fundamental countermeasure, andselecting an optimal emergency countermeasure for the fundamentalcountermeasure in the case where the determination result isaffirmative. In the present embodiment, combination of the fundamentalcountermeasure and the emergency countermeasure selected for thefundamental countermeasure will be referred to as “optimal combination”.The predicted worst performance value of the specific bottleneck even ifboth side effects of the fundamental countermeasure and the emergencytreatment in the optimal combination are taken into account is smallerthan the second threshold.

The plan information 319 is completed through the optimal combinationgeneration. The plan generation program 310 kicks the prioritycalculation program 312. The priority calculation program 312 calculatespriority of each of one or more plans indicated by the plan information319 (S1004). The priority calculation corresponds to preparation fordisplaying plans in descending order of the priority in screen display(S905). The priority is calculated on the basis of, for example, thefollowing policy, that is, at least one of (p1) to (p5). (p2) to (p5)are values which can be identified from the failure countermeasureinformation 320. Note that arrangement of (p1) to (p5) means descendingorder of weighting. For example, in the case where the number ofcountermeasures of the first plan is smaller than that of the secondplan, but an extent of influence of the first plan is larger than thatof the second plan, because the first plan is superior to the secondplan for the number of countermeasures with higher weighting, thepriority of the first plan may be higher than that of the second plan.The order of weighting does not have to be limited to (p1) to (p5).

-   (p1) The number of countermeasures. The number of countermeasures is    the number of countermeasures included in the plan. The priority of    the plan is higher for a smaller number of countermeasures.-   (p2) The extent of influence of the plan. The extent of influence of    the plan is an extent of influence of execution of the plan (for    example, the number of affected elements), and is, specifically, a    sum of one or more extents of influence 406 (for example, the number    of elements) corresponding to the one or more countermeasures in the    plan. The priority of the plan is higher for a narrower extent of    influence of the plan.-   (p3) The plan required time. The plan required time is time required    for executing the plan, and is, specifically, a sum of one or more    pieces of execution required time 407 (and start required time 404)    corresponding to the one or more countermeasures in the plan.-   (p4) The number of times of plan operation. The number of times of    plan operation is the number of times of operation performed in    execution of the plan, and is, specifically, a sum of one or more    numbers of times of operation 408 corresponding to the one or more    countermeasures in the plan.-   (p5) The plan execution cost. The plan execution cost is cost    required for executing the plan, and is, specifically, a sum of one    or more pieces of execution cost 409 corresponding to the one or    more countermeasures in the plan.

FIG. 11 illustrates flow of countermeasure type determination (S1002).Note that the first threshold and the second threshold described in thefollowing description are the first threshold and the second thresholdidentified by the countermeasure type determination program 308 from thethreshold information 316 using an ID of the specific bottleneck as akey. FIG. 15 will be referred to as appropriate in description of FIG.11.

The countermeasure type determination program 308 calculates TL (S1101).TL is time limit, and is, specifically, a time point elapsed from TP bytime limit (time indicated by the time limit information 321). TP is aplan generation time point, and is, specifically, for example, a timepoint at which plan generation (S904) is started. T0 is a failureoccurrence time point, and is, specifically, for example, a time pointat which a performance value exceeding the first threshold is measuredor a time point at which a failure is detected. TL is a reference to becompared with various types of time points in classification anddetermination of the countermeasure. The time limit or TL may be a valuemanually input by the administrator.

The countermeasure type determination program 308 calculates T2 (S1102).T2 is a performance degradation time point, and is, specifically, a timepoint at which a performance value following the predicted trend reachesthe second threshold. T2 is an x coordinate of an intersection of thepredicted trend and the second threshold.

The countermeasure type determination program 308 executes the followingS1103 to S1114 for each of all the countermeasures indicated in thefailure countermeasure information 320. S1103 to S1114 will be describedbelow using one countermeasure as an example. At this time, the onecountermeasure will be referred to as a “target countermeasure” forconvenience sake in the description of FIG. 11.

The countermeasure type determination program 308 calculates a2 (S1103).a2 is an example of an amount of change after the countermeasure isimplemented, and is, specifically, inclination obtained by inputting a1and y1 to the evaluation function 403 corresponding to the targetcountermeasure and the specific bottleneck. y1 is a y coordinatecorresponding to TP. In other words, a2 indicates inclination of theperformance value later than TP.

The countermeasure type determination program 308 calculates T1 (S1104).T1 is a time point at which the performance value is predicted to reachthe first threshold on the basis of C1(TP, y1) and a2. T1 is an xcoordinate at an intersection of a line 1502 whose inclination is a2 andwhich passes through C1(TP, y1), and the x axis. Therefore, unless a2 isa negative value, T1 is infinite (refer to a line 1503).

The countermeasure type determination program 308 calculates TD (S1105).TD is a start time point (a time point at which execution of the targetcountermeasure is expected to be started). TD is calculated on the basisof the start required time 404 corresponding to the targetcountermeasure and the specific bottleneck and TP. Specifically, forexample, in the case where the start required time 404 is a constant, TDis a time point elapsed from TP by the start required time 404. In thecase where the start required time 404 is a function, TD is a tentativetime point based on TP and the start required time 404. The startrequired time 404 being a function means, for example, the startrequired time 404 of the fundamental countermeasure changing accordingto the emergency countermeasure to be combined. Note that TD may be acountermeasure completion time point (a time point elapsed from TP bythe start required time 404 and the execution required time 407) inplace of the start time point.

The countermeasure type determination program 308 executes two or moretypes of determination among the following determination.

-   S1106: T1<TL? (Is T1 earlier than TL?)-   S1107: TD<T2? (Is TD earlier than T2?)-   S1108: TD<T2 and a2<a1? (Is TD earlier than T2 and is a2 smaller    than a1?)-   S1109: TD<TL or is the start required time 404 not a constant?

The countermeasure type determination program 308 classifies the targetcountermeasure into one of the “fundamental treatment (the emergencycountermeasure is unknown)”, the “fundamental treatment (the emergencycountermeasure is required)”, the “emergency treatment” and “ineffectivetreatment” on the basis of results of these determination (S1110 toS1113). The “fundamental treatment (the emergency countermeasure isunknown)” is fundamental treatment for which it is unknown whether ornot the emergency countermeasure is required. The “fundamental treatment(the emergency countermeasure is required)” is fundamental treatment forwhich the emergency countermeasure is required. The countermeasure typedetermination program 308 registers the classification result in thecountermeasure type information 318 (S1114). Specifically, thecountermeasure type determination program 308 adds a record to thecountermeasure type information 318 and registers a failure IDcorresponding to the target countermeasure, a countermeasure ID, name ofa countermeasure type of a classification destination, necessity of theemergency countermeasure and inclination after the countermeasure isimplemented (a2 calculated in S1103) in the added record.

The countermeasure type classification (determination in S1106 to S1109and S1110 to S1113 in accordance with the determination results)conforms to the following policy.

It is desirable that only the fundamental countermeasure is needed.

In case where there is no much time until the start time point of thefundamental countermeasure, it is desirable to buy time by executing theemergency countermeasure.

It is desirable to avoid a start time point of the fundamentalcountermeasure from being too late.

Case 1 to case 7 will be described below as examples.

Case 1 (FIG. 16)

T1>TL and a2>a1. In case 1, the target countermeasure is classified intothe “ineffective treatment” (the countermeasure type other than thefundamental treatment, the emergency treatment and the follow-uptreatment) (S1113), because the performance value does not become better(the performance value degrades).

Case 2 (FIG. 17)

T1>TL, a2<a1 and TD<T2. In case 2, the target countermeasure isclassified into the “emergency treatment” (S1112), because, while T1 isinfinite, a2 (inclination after the countermeasure is implemented) issmaller than a1 (inclination before the countermeasure is implemented(inclination of the trend 1501)), which means speed of degradation ofthe performance value can be reduced.

Case 3 (FIG. 18)

T1>TL, a2<a1 and TD>T2. In case 3, the target countermeasure isclassified into the “ineffective treatment” (S1113), because the targetcountermeasure cannot be started before the performance value reachesthe second threshold.

Case 4 (FIG. 19)

T1<TL and TD<T2. In case 4, the target countermeasure is classified intothe “fundamental treatment (the emergency countermeasure is unknown)”(S1110), because the target countermeasure can be started before theperformance value reaches the second threshold, and T1 comes earlierthan the time limit.

Case 5 (FIG. 20)

T1<TL, and T2<TD<TL. In case 5, the target countermeasure is classifiedinto the “fundamental treatment (the emergency countermeasure isrequired)” (S1111), because while T1 comes earlier than the time limit,the target countermeasure cannot be started before the performance valuereaches the second threshold.

Case 6 (FIG. 21)

T1<TL, TL<TD, and the start required time 404 of the targetcountermeasure is not a constant (for example, a function). In case 6,the target countermeasure is classified into the “fundamental treatment(the emergency countermeasure is required)” (S1111). The start requiredtime 404 being not a constant means that the start required time 404increases or decreases in accordance with the emergency countermeasureto be combined. Therefore, even if the start time point in accordancewith the tentative start required time is later than the time limit,there is a possibility that, actually, the start time point may beearlier than the time limit. Therefore, in case 6, the targetcountermeasure is classified into the “fundamental treatment (theemergency countermeasure is required)”.

Case 7 (FIG. 22)

T1<TL, TL<TD and the start required time 404 of the targetcountermeasure is a constant. In case 7, the target countermeasure isclassified into the “ineffective treatment” (S1113), because, unlikewith case 6, there is no possibility that, actually, the start timepoint may be earlier than the time limit.

FIG. 12 illustrates flow of the optimal combination generation (S1003).FIG. 23 will be referred to as appropriate in description of FIG. 12.

The optimal combination generation program 311 selects all thecountermeasures (the emergency countermeasures and the fundamentalcountermeasures) indicated by the countermeasure type information 318with reference to the countermeasure type information 318 as the resultof the countermeasure type determination (S1002) (S1201). The optimalcombination generation program 311 executes S1202 to S1209 for each ofthe fundamental countermeasures among the selected countermeasures.S1202 to S1209 will be described below using one fundamentalcountermeasure as an example. At that time, the one fundamentalcountermeasure will be referred to as a “target fundamentalcountermeasure” for convenience sake in description of FIG. 12 and FIG.13.

The optimal combination generation program 311 determines whether or notnecessity of the emergency countermeasure for the target fundamentalcountermeasure is unknown (S1202).

In the case where the determination result in S1202 is affirmative (inthe case where necessity of the emergency countermeasure 504corresponding to the target fundamental countermeasure is “unknown”)(S1202: Yes), the optimal combination generation program 311 identifiesF1 with reference to the side effect information 317 using the sideeffect ID 405 corresponding to the target fundamental countermeasure andthe specific element as a key (S1203). F1 is side effect detail 603(function) which satisfies the following (1203-1) and (1203-2).

-   (1203-1) Side effect detail 603 associated with the side effect ID    405 (601) corresponding to the target fundamental countermeasure and    the specific element.-   (1203-2) Side effect detail 603 associated with side effect outline    602 (for example, “increase in MPB busy rate”) with which the    performance value type (for example, MPB busy rate) corresponding to    the detected performance failure is compatible.

The optimal combination generation program 311 calculates SE (S1204). SEis a maximum width (performance value width) of the side effect for thetarget fundamental countermeasure. SE is a value obtained by inputtinga2 (inclination after countermeasure 505 corresponding to the targetfundamental countermeasure) and y2 (a y coordinate corresponding to theTD on the trend 1501) to F1 (SE may be a value depending on at least oneof a2 and y2).

The optimal combination generation program 311 calculates T3 (S1205). T3is a time point at which the performance value is predicted to reach thefirst threshold on the basis of C2(TD, y2) and a2. T3 is an x coordinateat an intersection of a line 2301 whose inclination is a2 and whichpasses through C2(TD, y2), and the x axis.

The optimal combination generation program 311 determines whether or noty2+SE<second threshold, and T3<TL (S1206).

In the case where the determination result in S1206 is affirmative(S1206: Yes), it is not necessary to implement the emergencycountermeasure before the target fundamental countermeasure isimplemented. Therefore, the optimal combination generation program 311executes S1209 which will be described later without executing emergencycountermeasure selection (S1207).

On the other hand, in the case where the determination result in S1206is negative (S1206: No), that is, in the case where y2+SE is equal to orlarger than the second threshold or T3 is equal to or larger than TL, itis necessary to implement the emergency countermeasure before the targetfundamental countermeasure is implemented. In a similar manner, in thecase where the determination result in S1202 is negative (in the casewhere the necessity of the emergency countermeasure 504 corresponding tothe target fundamental countermeasure is “required”) (S1202: No), it isnecessary to implement the emergency countermeasure before the targetfundamental countermeasure is implemented. Therefore, the optimalcombination generation program 311 kicks the emergency countermeasureselection program 323, and the emergency countermeasure selectionprogram 323 executes emergency countermeasure selection (S1207).

As a result of the emergency countermeasure selection, in the case wherethere is no selected emergency countermeasure (in the case where no IDof the emergency countermeasure is registered in a selection list whichwill be described later) (S1208: No), the optimal combination generationprogram 311 does not generate a plan including the target fundamentalcountermeasure. In other words, the target fundamental countermeasure iseventually treated as an ineffective countermeasure.

On the other hand, as a result of the emergency countermeasureselection, in the case where there are one or more selected emergencycountermeasures (S1208: Yes), the optimal combination generation program311 executes S1209 which will be described later.

In S1209, the optimal combination generation program 311 generates oneor more plans including at least the target fundamental countermeasureand registers each of the generated one or more plans in the planinformation 319. The plan generation is as follows when description isprovided using one plan as an example.

-   (1209-1) In the case where the emergency countermeasure selection is    not performed for the target fundamental countermeasure, the optimal    combination generation program 311 generates a plan including the    target fundamental countermeasure. In this case, a plan ID 801 and a    fundamental countermeasure ID 802 are registered in a record of the    plan information 319.-   (1209-2) In the case where the emergency countermeasure selection is    performed for the target fundamental countermeasure and there are    one or more emergency countermeasures, the optimal combination    generation program 311 generates a plan including the target    fundamental countermeasure and any one of the emergency    countermeasures selected for the target fundamental countermeasure.    In this case, a plan ID 801, a fundamental countermeasure ID 802 and    an emergency countermeasure ID 803 are registered in a record of the    plan information 319.-   (1209-3) In any of (1209-1) and (1209-2), in the case where a    follow-up countermeasure is associated with at least one of F1 and    F2 (which will be described later), the optimal combination    generation program 311 generates a plan further including the    follow-up countermeasure. In this case, in addition to the plan ID    801 and at least the fundamental countermeasure ID 802 among the    fundamental countermeasure ID 802 and the emergency countermeasure    ID 803, the follow-up countermeasure ID 804 of the above-described    associated follow-up countermeasure is registered in the record of    the plan information 319. Note that the follow-up countermeasure ID    may be registered in the side effect information 317, or the    management information 360 may include follow-up countermeasure    information (which is not illustrated), and the follow-up    countermeasure information may indicate the countermeasure ID 401,    the follow-up outline 604, or the like, for each of a plurality of    the follow-up countermeasures.

FIG. 13 illustrates flow of the emergency countermeasure selection(S1207). FIG. 24 to FIG. 27 will be referred to as appropriate indescription of FIG. 13.

The emergency countermeasure selection program 323 executes S1301 toS1312 for each of the emergency countermeasures selected in S1201. S1301to S1312 will be described below using one emergency countermeasure asan example. At that time, the one emergency countermeasure will bereferred to as a “target emergency countermeasure” for convenience sakein the description of FIG. 13.

The emergency countermeasure selection program 323 calculates TD1(S1301). TD1 is a new start time point of the target fundamentalcountermeasure, that is, a start time point of the target fundamentalcountermeasure after the target emergency countermeasure is implemented.TD1 is calculated using start required time 404 (a function or aconstant) corresponding to the target fundamental countermeasure and thespecific bottleneck and time corresponding to the target emergencycountermeasure and the specific bottleneck (the start required time 404,the execution required time 407 or a sum of these). Note that a valueused for calculating TD1 is not limited to time. For example, incalculation of TD1, the countermeasure ID 401 of the target emergencycountermeasure may be input to the start required time 404 (function)corresponding to the target fundamental countermeasure and the specificbottleneck.

The emergency countermeasure selection program 323 determines whether ornot TD2<TD1 (S1302). TD2 is a start time point corresponding to thetarget emergency countermeasure and the specific bottleneck, that is, atime point elapsed from TP by the start required time 404 correspondingto the target emergency countermeasure and the specific bottleneck. Inthe case where the determination result in S1302 is negative (S1302:No), the target emergency countermeasure is not selected as theemergency countermeasure to be combined with the target fundamentalcountermeasure, because the target emergency countermeasure cannot bestarted before the target fundamental countermeasure is implemented.

In the case where the determination result in S1302 is affirmative(S1302: Yes), the emergency countermeasure selection program 323identifies F2 (S1303). F2 is side effect detail 603 (function) whichsatisfies the following (1303-1) and (1303-2).

-   (1303-1) The side effect detail 603 associated with the side effect    ID 405 (601) corresponding to the target emergency countermeasure    and the specific element.-   (1303-2) The side effect detail 603 associated with side effect    outline 602 (for example, “increase in MPB busy rate”) with which    the performance value type (for example, MPB busy rate)    corresponding to the detected performance failure is compatible.

The emergency countermeasure selection program 323 calculates SE1(S1304). SE1 is a maximum width (performance value width) of a sideeffect for the target emergency countermeasure, and is a value obtainedby inputting a3 and y3 in F2 (SE1 may be a value depending on at leastone of a3 and y3). a3 is inclination after countermeasure 505corresponding to the target emergency countermeasure. y3 is a ycoordinate corresponding to TD2 on the trend 1501.

The emergency countermeasure selection program 323 determines whether ornot y3+SE1<second threshold (S1305). In the case where the determinationresult in S1305 is negative (S1305: No), the target emergencycountermeasure is not selected as the emergency countermeasure to becombined with the target fundamental countermeasure, because there is arisk that the performance value of the specific bottleneck may reach thesecond threshold due to the side effect of the target emergencycountermeasure.

In the case where the determination result in S1305 is affirmative(S1305: Yes), the emergency countermeasure selection program 323determines whether or not F1 has been selected (S1306).

In the case where S1202: No, because S1203 is skipped, F1 has not beenselected (S1306: No). Therefore, the emergency countermeasure selectionprogram 323 selects F1 (S1307). S1307 may be the same as S1203. On theother hand, in the case where S1202: Yes, because S1203 is executed, F1has been selected (S1306: Yes). Therefore, S1307 is skipped.

After S1306: Yes, or S1307, the emergency countermeasure selectionprogram 323 calculates SE1′ and SE2 (S1308). Specifically, the followingis performed.

-   (S1308-1) The emergency countermeasure selection program 323    calculates a2′. a2′ is an example of an amount of change after    combination of the target emergency countermeasure and the target    fundamental countermeasure is implemented, and is, specifically,    inclination obtained by inputting inclination after the emergency    countermeasure is implemented (that is, a2 of the target emergency    countermeasure (inclination after countermeasure 505 in FIG. 5)) and    y4 to the evaluation function 403 of the target fundamental    countermeasure. y4 is a y coordinate corresponding to TD1 on a line    (line having inclination a3) 2401 which passes through C3(TD2, y3).-   (S1308-2) The emergency countermeasure selection program 323    calculates SE1′. SE1′ is a maximum width (performance value width)    of the side effect for combination of the target emergency    countermeasure and the target fundamental countermeasure, and is a    value obtained by inputting a2′ and y4 to F2 (SE1′ may be a value    depending on at least one of a2′ and y4). As described above, F2 is    side effect detail 603 for the target emergency countermeasure (more    specifically, as described above, side effect detail 603 (function)    which satisfies (1301-1) and (1301-2)).-   (S1308-3) The emergency countermeasure selection program 323    calculates SE2. SE2 is a maximum width (performance value width) of    the side effect for the target fundamental countermeasure after the    target emergency countermeasure is implemented, and is a value    obtained by inputting a2′ and y4 to F1 (SE2 may be a value depending    on at least one of a2′ and y4). As described above, F1 is side    effect detail 603 for the target fundamental countermeasure (more    specifically, as described above, side effect detail 603 (function)    which satisfies (1203-1) and (1203-2)).

The emergency countermeasure selection program 323 determines whether ornot y4+SE1′+SE2<second threshold (S1309). In the case where thedetermination result in S1309 is negative (S1309: No), the targetemergency countermeasure is not selected as the emergency countermeasureto be combined with the target fundamental countermeasure, because, ifboth the target emergency countermeasure and the target fundamentalcountermeasure are implemented, there is a risk that the performancevalue of the specific bottleneck may reach the second threshold due tothe side effect.

In the case where the determination result in S1309 is affirmative(S1309: Yes), the emergency countermeasure selection program 323calculates T4 (S1310). T4 is a time point at which the performance valueis predicted to reach the first threshold on the basis of C4(TD1, y4)and a2′. T4 is an x coordinate at an intersection of a line 2402 whoseinclination is a2′ and which passes through C4(TD1, y4) and the x axis.

The emergency countermeasure selection program 323 determines whether ornot T4<TL (S1311). In the case where the determination result in S1311is negative (S1311: No), the target emergency countermeasure is notselected as the emergency countermeasure to be combined with the targetfundamental countermeasure, because it is not expected that the failureis recovered before the time limit.

In the case where the determination result in S1311 is affirmative(S1311: Yes), the emergency countermeasure selection program 323registers the target emergency countermeasure (for example, acountermeasure ID of the target emergency countermeasure) in a selectionlist (which is not illustrated) (S1312). In other words, the emergencycountermeasure selection program 323 selects the target emergencycountermeasure as the emergency countermeasure to be combined with thetarget fundamental countermeasure.

As described with reference to FIGS. 10 to 13, in the presentembodiment, the management computer 201 roughly classifies each of theplurality of countermeasures into any countermeasure type in thecountermeasure type determination (first stage processing) anddetermines whether the classified fundamental countermeasure is made afundamental countermeasure to be included in a plan or which emergencycountermeasure is to be combined with the fundamental countermeasure inthe optimal combination generation (second stage processing). Throughsuch two-stage processing, for example, the following operation effectscan be expected.

That is, there can be a countermeasure which causes a side effect in thespecific bottleneck among the plurality of countermeasures. There is apossibility that a performance value of the specific bottleneck maydegrade due to the side effect upon start of the countermeasure orthereafter. However, in the case where the countermeasure is thefundamental countermeasure, a start time point of the fundamentalcountermeasure changes according to whether or not the emergencycountermeasure is combined with the fundamental countermeasure and, ifthe emergency countermeasure is combined, which emergency countermeasureis to be combined. In view of such an environment, the managementcomputer 201 roughly (tentatively) classifies the countermeasure intoany countermeasure type using a time point such as TP as a referencetime point in the countermeasure type determination, and determineswhether or not the countermeasure is made a countermeasure to beincluded in the plan using a predicted start time point of thecountermeasure as a reference time point in the subsequent optimalcombination. By this means, it is possible to present an appropriateplan for recovering from the failure.

FIG. 14 illustrates flow of screen display (S905) and plan selection(S906).

In screen display (S905), the screen display program 313 receives thepriority calculated in S1004 for each of the one or more plans indicatedin the plan information 319 from the priority calculation program 312(S1401). The screen display program 313 displays the plan selectionscreen 150 (see FIG. 1) in which one or more plans are arranged indescending order of the priority (S1402). In S1402, the screen displayprogram 313 may also display one or more pieces of priority respectivelycorresponding to the one or more plans at the plan selection screen 150.

In plan selection (S906), the screen display program 313 receivesselection of a plan (S1403). The screen display program 313 transmitsthe selected plan (for example, a plan ID) to the effectivenessrecalculation program 322.

The effectiveness recalculation program 322 then recalculateseffectiveness of the selected plan. As described above, therecalculation means performing processing similar to processingperformed in plan generation (S904) for generating the selected plan onthe basis of a plan selection time point TC (see FIG. 28). In otherwords, processing from FIG. 11 to FIG. 13 is performed for thefundamental countermeasure (and the emergency countermeasure) includedin the selected plan assuming that TP=TC.

The meaning of the effectiveness recalculation is, for example, asfollows. In other words, because time ticks away, a countermeasure in atleast one plan (for example, a plan selected by the administrator) amongthe generated plans (displayed plans) does not necessarily remaineffective at a certain time point (for example, a time point at which aplan is selected by the administrator) TC after the plan generation,because TC may be later than at least one of TD, TD1 and TD2. In thepresent embodiment, effectiveness recalculation which is processingsimilar to processing from FIG. 11 to FIG. 13 is performed on the basisof TP=TC only for at least one plan (for example, the plan selected bythe administrator) among the generated plans (displayed plans). By thismeans, it is possible to improve certainty that the selected andexecuted plan is an effective plan (a plan which does not include anineffective countermeasure).

Note that TC may be a time point following a predetermined period (forexample, a time point following a period of one minute after the plan isdisplayed) in place of a time point when the plan is selected. Thepredetermined period may be a period configured by the administrator.

The plan execution (S909) is, for example, as follows. It is assumedthat a plan to be executed includes the emergency countermeasure, thefollow-up countermeasure and the fundamental countermeasure. The planexecution program 324 executes follow-up in accordance with thefollow-up countermeasure at a plan execution start time point or afterthe plan execution start time point (for example, before the fundamentalcountermeasure is started). The plan execution program 324 finishes thefollow-up at a follow-up releasing trigger 605 associated with thefollow-up countermeasure. The program execution program can start thefundamental countermeasure in the plan to be executed in the case wherethe performance value of the follow-up target satisfies predeterminedconditions in the follow-up.

While an embodiment has been described above, this is an example forexplaining the present invention and is not intended to limit the scopeof the present invention only to the embodiment. The present inventioncan be implemented in other various forms.

For example, the plan execution program 324 may execute a plan with thehighest priority among plans including at least one of the emergencycountermeasure and the follow-up countermeasure without receivingselection of a plan from the administrator (further, without display ofthe plan selection screen 150). Then, in the case where conditionsmeaning a state where any fundamental countermeasure can be taken aresatisfied (for example, in the case where the performance value of thespecific bottleneck satisfies predetermined conditions), the screendisplay program 313 may display the above-described one or moregenerated plans including a plan in which at least one of the emergencycountermeasure and the follow-up countermeasure has been executed, andthe plan execution program 324 may receive selection of one of the oneor more plans from the administrator. If the selected plan includes atleast one of the emergency countermeasure and the follow-upcountermeasure, the plan execution program 324 may execute thecountermeasure, or, even if the selected plan includes at least one ofthe emergency countermeasure and the follow-up countermeasure, the planexecution program 324 may execute the fundamental countermeasure withoutexecuting the countermeasure. In the case where an urgent failureoccurs, because at least one of the emergency countermeasure and thefollow-up countermeasure is automatically executed without a plan beingselected by the administrator, it is possible to reduce management load.

REFERENCE SIGNS LIST

-   201: Management computer

The invention claimed is:
 1. A management system comprising: aninterface coupled to a computer system including a plurality of elementswhose performance values are monitored; a memory which stores managementinformation including information indicating relationship between aplurality of countermeasure attributes and a plurality of bottlenecksfor each of a plurality of countermeasures; and a processor coupled tothe interface and the memory, wherein each of the plurality ofbottlenecks is an element which can be a cause of a failure, and theprocessor is configured to: (A) generate one or more plans on the basisof history of a performance value of a specific bottleneck which is anelement identified as a cause of a detected failure, a countermeasureattribute corresponding to the specific bottleneck among the pluralityof countermeasure attributes of the plurality of countermeasures, andthe management information; and (B) display the generated one or moreplans, each of the displayed one or more plans including one or morecountermeasures, and in each of the displayed one or more plans, each ofthe one or more countermeasures being classified into one of a pluralityof countermeasure types.
 2. The management system according to claim 1,wherein the plurality of countermeasure types include fundamentaltreatment type countermeasures with which the performance value of thespecific bottleneck is expected to be returned to equal to or smallerthan a first threshold of the performance value until time limit, andemergency treatment type countermeasures with which an amount of changeof the performance value of the specific bottleneck is expected to bereduced, and a plurality of countermeasure attributes respectivelyassociated with the plurality of countermeasures include (x1) change ofa performance value expected for the respective countermeasure, and (x2)at least one of start required time which is time required until therespective countermeasure is started and execution required time whichis time from when the respective countermeasure is started until whenthe respective countermeasure is finished.
 3. The management systemaccording to claim 2, wherein an emergency treatment type countermeasuresatisfies that an amount of change after the emergency treatment typecountermeasure is implemented which is an amount of change of aperformance value in accordance with (x1) corresponding to the emergencytreatment type countermeasure is smaller than an amount of change beforethe emergency treatment type countermeasure is implemented which is anamount of change of an emergency treatment performance value inaccordance with history of the emergency treatment performance value ofthe specific bottleneck, and a time point in accordance with (x2)corresponding to the emergency treatment type countermeasure is earlierthan a time point at which the emergency treatment performance valuedegrades which is a time point at which the emergency treatmentperformance value of the specific bottleneck is predicted to reach asecond threshold on the basis of the amount of change before theemergency treatment type countermeasure is implemented, and the secondthreshold is a threshold meaning that the emergency treatmentperformance value is worse than the first threshold.
 4. The managementsystem according to claim 2, wherein the plurality of countermeasuretypes further include follow-up treatment, the management informationindicates a secondary effect of a performance value for at least onecountermeasure and a follow-up countermeasure which is a countermeasureclassified into the follow-up treatment for at least one secondaryeffect, and a plan including a countermeasure associated with asecondary effect associated with the follow-up countermeasure among thedisplayed one or more plans further includes the follow-upcountermeasure.
 5. The management system according to claim 2, wherein,the processor is configured to, after (A),automatically execute anemergency treatment type countermeasure in any plan without executing(B) or without receiving plan selection, and in the case whereconditions meaning a state where any fundamental treatment typecountermeasure can be taken are satisfied, execute (B).
 6. Themanagement system according to claim 2, wherein each countermeasureincluded in the displayed one or more plans is an effectivecountermeasure which can be included in any plan on the basis of aresult of comparison between the time limit and a time point inaccordance with (x2) of the displayed countermeasure.
 7. The managementsystem according to claim 2, wherein, for at least one countermeasureamong the plurality of countermeasures, the management informationindicates a secondary effect of the respective performance value, in aplan including a countermeasure associated with the secondary effect, apredicted performance value predicted in accordance with secondaryeffects of all countermeasures included in the plan, which is aperformance value at a time point in accordance with (x2) of acountermeasure classified into the fundamental treatment is smaller thana second threshold.
 8. The management system according to claim 7,wherein the management information further indicates a follow-upcountermeasure for at least one secondary effect, and among thedisplayed one or more plans, a plan including a countermeasureassociated with the secondary effect associated with the follow-upcountermeasure further includes the follow-up countermeasure.
 9. Themanagement system according to claim 2, wherein the managementinformation indicates a secondary effect of a performance value for atleast one countermeasure among the plurality of countermeasures, theprocessor is configured to: in (A), (a1) classify each of the pluralityof countermeasures as one of the fundamental treatment type, theemergency treatment type or an ineffective treatment type which does notcorrespond to either the fundamental treatment and the emergencytreatment types of countermeasures; (a2) for each countermeasure whichis classified as a fundamental treatment type of countermeasure in (a1);(a21) identify a secondary effect associated with the countermeasurefrom the management information; (a22) determine whether or not tocombine an emergency treatment type countermeasure with thecountermeasure on the basis of the identified secondary effect; (a23) inthe case where a determination result in (a22) is negative, generate aplan including the countermeasure; and (a24) in the case where adetermination result in (a22) is positive, generate one or more planseach including the countermeasure and the emergency treatment typecountermeasure, and each of the one or more plans generated in (A) is aplan including at least a countermeasure classified as the fundamentaltreatment type.
 10. The management system according to claim 9, wherein,for each respective countermeasure of the fundamental treatment typecountermeasures, determination in (a22) is determination as to whetheror not a performance value upon start is smaller than a secondthreshold, and a time point after the respective fundamental treatmenttype countermeasure is implemented is earlier than the time limit, theperformance value upon start is a performance value predicted inaccordance with a secondary effect associated with the respectivefundamental treatment type countermeasure, and is a performance value ata time point in accordance with (x2) of the respective fundamentaltreatment type countermeasure, the second threshold is a thresholdmeaning that the performance value is worse than the first threshold,and the time point after the respective fundamental treatment typecountermeasure is implemented is a time point at which the performancevalue is predicted to be returned to equal to or smaller than the firstthreshold on the basis of an amount of change of the performance valuein accordance with (x1) corresponding to the respective fundamentaltreatment type countermeasure and a time point in accordance with (x2)of the respective fundamental treatment type countermeasure.
 11. Themanagement system according to claim 10, wherein the processor isconfigured to, in (a24), for each respective countermeasure of theemergency treatment type countermeasures, predict, on the basis of asecondary effect associated with the respective emergency treatment typecountermeasure, a secondary effect associated with a fundamentaltreatment type countermeasure and an amount of change before therespective emergency treatment type countermeasure is implemented whichis an amount of change of the performance value in accordance withhistory of the performance value of the specific bottleneck, aperformance value at a time point in accordance with (x2) of thefundamental treatment type countermeasure, and in the case where thepredicted performance value is smaller than the second threshold, selectthe respective emergency treatment type countermeasure as an emergencytreatment type countermeasure to be included in a plan along with thefundamental treatment type countermeasure.
 12. The management systemaccording to claim 1, wherein, in each of the displayed one or moreplans, each of the one or more countermeasures is classified into one ofthe plurality of countermeasure types on the basis of a result ofcomparison between an amount of change after the respectivecountermeasure is implemented which is an amount of change of arespective performance value in accordance with change of a performancevalue expected for the respective countermeasure corresponding to therespective countermeasure and an amount of change before the respectivecountermeasure is implemented which is an amount of change of arespective performance value in accordance with history of therespective performance value of the specific bottleneck, and a result ofcomparison between a time point in accordance with at least one of startrequired time which is time required until the respective countermeasureis started and execution required time which is time from when therespective countermeasure is started until when the respectivecountermeasure is finished corresponding to the respectivecountermeasure and a time point at which the respective performancevalue degrades which is a time point at which the respective performancevalue of the specific bottleneck is predicted to reach a secondthreshold on the basis of the amount of change before the respectivecountermeasure is implemented, and the second threshold is a thresholdmeaning that the respective performance value is worse than the firstthreshold.
 13. The management system according to claim 1, wherein thegenerated one or more plans is a plan generated on the basis of a timepoint defined as a plan generation time point, and the processor isconfigured to: execute effectiveness determination of determining, onthe basis of a time point at which any plan among the displayed one ormore plans is selected or a time point designated in advance, whether ornot each of the displayed one or more plans or the selected plan remainseffective; and in the case where a result of the effectivenessdetermination is affirmative, execute the plan selected after theeffectiveness determination or the selected plan before theeffectiveness determination.
 14. A management method of a computersystem including a plurality of elements whose performance values aremonitored, the method comprising: (A) generating one or more plans onthe basis of history of a performance value of a specific bottleneckwhich is an element identified as a cause of a detected failure,management information including information indicating relationshipbetween a plurality of countermeasure attributes and a plurality ofbottlenecks for each of a plurality of countermeasures, and acountermeasure attribute corresponding to the specific bottleneck amongthe plurality of countermeasure attributes of the plurality ofcountermeasures, and (B) displaying the generated one or more plans,each of the displayed one or more plans including one or morecountermeasures, and in each of the displayed one or more plans, each ofthe one or more countermeasures being classified into one of a pluralityof countermeasure types.
 15. A non-transitory recording medium readableby a processor having a computer program for execution by a computercoupled to a computer system including a plurality of elements whoseperformance values are monitored, the computer program causing thecomputer to execute: (A) generation of one or more plans on the basis ofhistory of a performance value of a specific bottleneck which is anelement identified as a cause of a detected failure, managementinformation including information indicating relationship between aplurality of countermeasure attributes and a plurality of bottlenecksfor each of a plurality of countermeasures, and a countermeasureattribute corresponding to the specific bottleneck among the pluralityof countermeasure attributes of the plurality of countermeasures, and(B) display of the generated one or more plans, each of the displayedone or more plans including one or more countermeasures, and in each ofthe displayed one or more plans, each of the one or more countermeasuresbeing classified into one of a plurality of countermeasure types.